During the second week of each unit, we’ll “walk through” a basic research workflow, or data analysis process, modeled after the Data-Intensive Research Workflow from Learning Analytics Goes to School (Krumm et al., 2018):
Figure 2.2 Steps of Data-Intensive Research Workflow
Each walkthrough will focus on a basic analysis using text mining techniques that you’ll be expected to reproduce, and apply to a new research question next week, using the provided dataset or dataset of your own choosing.
This week, we will focus on analysis of open-ended survey items from an evaluation of the North Carolina Department of Public Instruction (NCDPI) online professional development offered as part of the state’s Race to the Top efforts. For more information about the Race to the Top evaluation work, visit https://cerenc.org/rttt-evaluation/.
For Unit 1, our focus will be on getting our text “tidy” so we can perform some basic word counts, look at words that occur at a higher rate in a group of documents, and examine words that are unique to those document groups. Specifically, the Unit 1 Walkthrough will cover the following workflow topics:
While we won’t investigate approaches to Model our data until Unit 3 when we learn about topic models, we will be developing “data products” next week to Communicate our findings and insights.
Prior to analysis, it’s critical to understand the context and data sources available so you can formulate useful questions that can be feasibly addressed by your data. For this section, we’ll focus on the following topics:
RTT Online Professional Development Evaluation
North Carolina was one of 12 recipients of the 2010 federal Race to the Top (RttT) grants, bringing nearly $400 million to the state’s public school system. Over the course of four years, NC’s RttT coordinated a set of activities and policy reforms designed to collectively improve the performances of students, teachers, leaders, and schools.
The North Carolina Race to the Top (RttT) proposal (North Carolina Office of the Governor, 2010) specifies that the state’s Professional Development Initiative will focus on the “use of e- learning tools to meet the professional development needs of teachers, schools, and districts” (p. 191). It points to research demonstrating that “well-designed and -implemented online professional development programs are not only valued by teachers but also positively impact classroom practices and student learning.”
Data Source & Analysis
The evaluation used a wide range of data sources including interviews, document review, site analytics, and surveys, which we’ll focus on for this walkthrough. Survey protocols were designed in cooperation with NCDPI to systematically collect information about local professional development, state-level supports, use of available RttT professional development resources, and organizational and classroom practices in the schools, which will serve as a baseline to assess changes over the period of the North Carolina RttT initiatives.
Quantitative analyses focused primarily on descriptive analysis of item-level responses. In addition, quantitative data from these surveys were analyzed to examine patterns in responses by participants’ role, event type (e.g., module, webinar, resource), and region. Responses to open-ended survey items of the Online Resources Survey were manually coded by their relation to each Learning Forward professional development standard.
Note that the dataset we’ll be using for analysis in this walkthrough is exported as is from Qualtrics with personal identifiers, select demographics, metadata, and closed-ended responses removed.
Summary of Findings
Approximately half of the state’s educators completed at least one online module by the end of the 2011-12 school year. Overall, most participants agreed that the webinars and modules were relevant to their professional development needs, though some content was redundant with prior PD activities and not always content- or grade-specific, and some modules did not meet national standards. Most online modules were completed independently and not in Professional Learning Community groups.
A common theme from focus groups and open-ended survey responses was the convenience of online professional development. One teacher in a focus group stated, “I liked the format. And the way that it was given, it was at your own pace, which works well for our schedules…” Educators also frequently cited that the information and resources provided through the modules improved their understanding of the new standards and the teacher evaluation process. Webinar participants appreciated the useful, updated information presented through a combination of PowerPoint slides and video clips.
While the majority of educators have indicated their satisfaction with these resources, the findings suggest that the use of these resources at both the state and local level was not wholly consistent with national standards for online professional development. Many LEAs likely needed additional guidance, training, support, technology tools, and/or content resources to ensure that local efforts contribute to the quality of the experiences for educators and that the vision for online professional development outlined in the state’s RttT proposal is realized and can be sustained beyond RttT.
The State’s progress on designing and implementing online professional development was originally guided by the following (very) general evaluation questions:
For this walkthrough, we’ll use text mining to complement prior qualitative analyses conducted as part of the RttT Evaluation by examining responses to open-ended questions on the RttT Online PD Survey administered to over 15,000 NC educators.
Our (very) specific questions of interest for this walkthrough are:
Finally, one overarching question we’ll explore throughout this course, and that Silge and Robinson (2018) identify as a central question to text mining and natural language processing, is:
How do we to quantify what a document or collection of documents is about?
As highlighted in Chapter 6 of Data Science in Education Using R (DSIEUR), one of the first steps of every workflow should be to set up a “Project” within RStudio. This will be your “home” for any files and code used or created in Unit 1. Open RStudio and follow these steps from DESIUR 6.6 to create a Project for Unit 1:
Create New File
Now that you have a Project to store .R scripts that you create as you work through this unit, let’s create our first .R script:
Finally, using your newly created R script, type the following code to load the packages we installed last week and that we’ll be needing for this walkthrough.
library(tidyverse)
## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
## ✓ ggplot2 3.3.3 ✓ purrr 0.3.4
## ✓ tibble 3.0.5 ✓ dplyr 1.0.2
## ✓ tidyr 1.1.2 ✓ stringr 1.4.0
## ✓ readr 1.4.0 ✓ forcats 0.5.0
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag() masks stats::lag()
library(tidytext)
At the end of this week, I’ll ask that you share with me your r script as evidence that you have complete the walkthrough. Although I highly recommend that that you manually type the code shared throughout this walkthrough, for large blocks of text it may be easier to cut and paste.
In general, data wrangling involves some combination of cleaning, reshaping, transforming, and merging data (Wickham & Grolemund, 2017). The importance of data wrangling is difficult to overstate, as it involves the initial steps of going from raw data to a dataset that can be explored and modeled (Krumm et al, 2018).
dplyr package to view, rename, select, slice, and filter our data in preparation for analysis.tidytext package to both “tidy” and tokenize our text in order to create a data frame to use for analysis.The Reading Data section introduces the following functions for reading data into R and inspecting it’s contents:
dplyr::read_csv() Reading .csv files into R.base::print() View your data frame in the Console Paneutils::view() View your data frame in the Source Panetibble::glimpse() Like print, but transposed so you can see all columnsutils::head() View the first 6 rows of your data.utils::tail() View last 6 rows of your data.Remember, the name before the double colon indicates the package the function comes from. For example, read_csv comes from the `readr`` package.
To get started, we need to import, or “read”, our data into R. The function used to import your data will depend on the file format of the data you are trying to import.
opd_survey.csv file we’ll be using for this Unit from our NCSU Moodle course site.Now let’s read our data into our Environment and assign it to a variable name so we can work with it like any other object in R.
opd_survey <- read_csv("data/opd_survey.csv")
## Warning: Duplicated column names deduplicated: 'Resource' => 'Resource_1' [10],
## 'Resource_10_TEXT' => 'Resource_10_TEXT_1' [11], 'Q16' => 'Q16_1' [12]
##
## ── Column specification ────────────────────────────────────────────────────────
## cols(
## RecordedDate = col_character(),
## ResponseId = col_character(),
## Role = col_character(),
## Q14 = col_character(),
## Q16 = col_character(),
## Resource = col_character(),
## Resource_8_TEXT = col_character(),
## Resource_9_TEXT = col_character(),
## Resource_10_TEXT = col_character(),
## Resource_1 = col_character(),
## Resource_10_TEXT_1 = col_character(),
## Q16_1 = col_character(),
## Q16_9_TEXT = col_character(),
## Q19 = col_character(),
## Q21 = col_character(),
## Delta = col_character(),
## Q26 = col_character(),
## Q37 = col_character(),
## Q8 = col_character()
## )
Notice that read_csv() dealt with the issues of duplicate column names for us!!
If you happen to run into issues with data import, RStudio as has an Import Dataset feature for a point and click approach to adding data to your environment. Be sure to pay attention to the
Once your data is in R, there are many different ways you can view it. Give each of the following at try:
# enter the name of your data frame and view directly in console
opd_survey
# view your data frame transposed so your can see every column and the first few entries
glimpse(opd_survey)
# look at just the first six entries
head(opd_survey)
# or the last six entries
tail(opd_survey)
# view the names of your variables or columns
names(opd_survey)
# or view in source pane
view(opd_survey)
In addition to reading data from your project folder, you can also write data back to a folder. The readr package has an intuitively named write_csv() function for doing just that.
Using the following code to create a copy of the opd_survey.csv file in your data folder from the opd_survey data frame you created:
write_csv(opd_survey, "data/opd_survey_copy.csv")
Note that the first argument is the data frame you created earlier and the second argument is the file name you plan to give it, including (if necessary) the file path for where it should go.
Throughout this walkthrough, you will be asked to respond to questions or short tasks to check your comprehension of the content covered. For section 2a. Read, View, and Write Data, please respond to these questions by commenting out a line or lines in your R script like so:
# 1. What argument would you add to `read_csv()` if my file did not not have column names or headers?
# I would need to add the ____ argument and set it to equal ____ to prevent R from setting the first row as column names.
read_csv() if my file did not not have column names or headers? You can type ?read_csv to get help on this function or check this handy cheatsheet for the readr package from the readr website at https://readr.tidyverse.org/index.htmlread_csv() always expects and what happens if you don’t include in quotes?view() compared to other functions for viewing your data?write_csv(opd_survey, "opd_survey_copy.csv") and just specify the file name instead including the folder?As you’ve probably already noticed from viewing our dataset, we clearly have more data than we need to answer our rather basic research question. For this part of our workflow we focus on the following functions from the dplyr package for wrangling our data:
dplyr functions
select() picks variables based on their names.slice() lets you select, remove, and duplicate rows.rename() changes the names of individual variables using new_name = old_name syntaxfilter() picks cases, or rows, based on their values in a specified column.stats functions
na.omit() a handy little function from the stats package for removing rows with missing values, i.e. NA.To begin, let’s select() Role, Resources, and Q21 columns and store as new data frame since those respectively pertain to educator role, OPD resource they are evaluating, and, as illustrated by the second row,
opd_selected <- select(opd_survey, Role, Resource, Q21)
Notice that like the bulk of all tidyverse functions, the first input select() expects is a data frame, followed by the columns you’d like to select.
Let’s take a look at our newly created data frame that should have dramatically fewer variables:
head(opd_selected)
## # A tibble: 6 x 3
## Role Resource Q21
## <chr> <chr> <chr>
## 1 "What is your role within you… "Please indicate the online pro… "How are you …
## 2 "{\"ImportId\":\"QID2\"}" "{\"ImportId\":\"QID3\"}" "{\"ImportId\…
## 3 "Central Office Staff (e.g. S… "Summer Institute/RESA PowerPoi… <NA>
## 4 "Central Office Staff (e.g. S… "Online Learning Module (e.g. C… "Collecting d…
## 5 "School Support Staff (e.g. C… "Online Learning Module (e.g. C… <NA>
## 6 "School Support Staff (e.g. C… "Calendar" "time analysi…
Notice that Q21 is not a terribly informative variable name. Let’s now take our opd_selected data frame and use the rename() function along with the = assignment operator introduced last week to change the name from Q21 to “text” and save it as opd_renamed.
This naming is somewhat intentional because not only is it the text we are interested in analyzing, but also mirrors the naming conventions in our [Text Mining with R]https://www.tidytextmining.com/tidytext.html course book and will make it easier to follow the examples there.
opd_renamed <- rename(opd_selected, text = Q21)
Now let’s deal with the legacy rows that Qualtrics outputs by default, which are effectively 3 sets of headers. We will use the slice() function, which is basically the same as the select() function but for rows instead of columns, to carve out those two rows.
opd_sliced <- slice(opd_renamed, -1, -2) # the - sign indicates to NOT keep rows 1 and 2
head(opd_sliced)
## # A tibble: 6 x 3
## Role Resource text
## <chr> <chr> <chr>
## 1 Central Office Staff (e.… Summer Institute/RESA Powe… <NA>
## 2 Central Office Staff (e.… Online Learning Module (e.… Collecting district fee…
## 3 School Support Staff (e.… Online Learning Module (e.… <NA>
## 4 School Support Staff (e.… Calendar time analysis
## 5 Teacher Live Webinar incorporating into all …
## 6 Teacher Online Learning Module (e.… Answering evaluation qu…
Now let’s take our opd_sliced and remove any rows that are missing data, as indicated by an NA.
opd_complete <- na.omit(opd_sliced)
Finally, since we are only interested in the feedback from teachers, let’s also filter our dataset for only participants who indicated their Role as “Teacher”.
opd_teacher <- filter(opd_complete, Role == "Teacher")
head(opd_teacher)
## # A tibble: 6 x 3
## Role Resource text
## <chr> <chr> <chr>
## 1 Teach… Live Webinar "incorporating into all content a…
## 2 Teach… Online Learning Module (e.g. Call f… "Answering evaluation questions."
## 3 Teach… Online Learning Module (e.g. Call f… "Use the online module to reflect…
## 4 Teach… Online Learning Module (e.g. Call f… "Preparing for Common core"
## 5 Teach… Online Learning Module (e.g. Call f… "gain knowledge"
## 6 Teach… Summer Institute/RESA PowerPoint Pr… "informing instructional practice…
That was a lot of code we just wrote to end up with opd_teacher. Let’s review:
opd_selected <- select(opd_survey, Role, Resource, Q21)
opd_renamed <- rename(opd_selected, text = Q21)
opd_sliced <- slice(opd_renamed, -1, -2)
opd_complete <- na.omit(opd_sliced)
opd_teacher <- filter(opd_complete, Role == "Teacher")
Note that we could have reused opd_teacher and simply overwritten it each time to prevent creating new objects:
opd_teacher <- select(opd_survey, Role, Resource, Q21)
opd_teacher <- rename(opd_teacher, text = Q21)
opd_teacher <- slice(opd_teacher, -1, -2)
opd_teacher <- na.omit(opd_teacher)
opd_teacher <- filter(opd_teacher, Role == "Teacher")
Fortunately, we can use the Pipe Operator %>% introduced in Chapter 6 of Data Science in Education Using R (DSIEUR) to dramatically simplify these cleaning steps and reduce the code written
opd_teacher <- opd_survey %>%
select(Role, Resource, Q21) %>%
rename(text = Q21) %>%
slice(-1, -2) %>%
na.omit() %>%
filter(Role == "Teacher")
head(opd_teacher)
## # A tibble: 6 x 3
## Role Resource text
## <chr> <chr> <chr>
## 1 Teach… Live Webinar "incorporating into all content a…
## 2 Teach… Online Learning Module (e.g. Call f… "Answering evaluation questions."
## 3 Teach… Online Learning Module (e.g. Call f… "Use the online module to reflect…
## 4 Teach… Online Learning Module (e.g. Call f… "Preparing for Common core"
## 5 Teach… Online Learning Module (e.g. Call f… "gain knowledge"
## 6 Teach… Summer Institute/RESA PowerPoint Pr… "informing instructional practice…
Our dataset is now ready to be tidied!!!
opd_benefits for later use.For this part of our workflow we focus on the following functions from the tidytext and dplyr packages respectively:
unnest_tokens() splits a column into tokensanti_join() returns all rows from x without a match in y.Not surprisingly, the Tidyverse set of packages including packages like dplyr adhere “tidy” data principles (Wickham 2014). Tidy data has a specific structure:
Why would this data be considered “untidy”?
Text data, by it’s very nature is ESPECIALLY untidy. In Chapter 1 of Text Mining with R, Silge and Robinson define the tidy text format as
a table with one-token-per-row. A token is a meaningful unit of text, such as a word, that we are interested in using for analysis, and tokenization is the process of splitting text into tokens. This one-token-per-row structure is in contrast to the ways text is often stored in current analyses, perhaps as strings or in a document-term matrix. For tidy text mining, the token that is stored in each row is most often a single word, but can also be an n-gram, sentence, or paragraph. In the tidytext package, we provide functionality to tokenize by commonly used units of text like these and convert to a one-term-per-row format.
In this section, our goals is to transform our opd_teacher data from this:
## # A tibble: 6 x 3
## Role Resource text
## <chr> <chr> <chr>
## 1 Teach… Live Webinar "incorporating into all content a…
## 2 Teach… Online Learning Module (e.g. Call f… "Answering evaluation questions."
## 3 Teach… Online Learning Module (e.g. Call f… "Use the online module to reflect…
## 4 Teach… Online Learning Module (e.g. Call f… "Preparing for Common core"
## 5 Teach… Online Learning Module (e.g. Call f… "gain knowledge"
## 6 Teach… Summer Institute/RESA PowerPoint Pr… "informing instructional practice…
to this:
## # A tibble: 6 x 3
## Role Resource word
## <chr> <chr> <chr>
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar into
## 3 Teacher Live Webinar all
## 4 Teacher Live Webinar content
## 5 Teacher Live Webinar areas
## 6 Teacher Live Webinar adn
In order to tidy our text, we need to break the text into individual tokens (a process called tokenization) and transform it to a tidy data structure. To do this, we use tidytext’s incredibly powerful unnest_tokens() function.
After all the work we did prepping our data, this is going to feel a little anticlimactic.
Let’s go ahead and tidy our text and save it as opd_tidy:
opd_tidy <- unnest_tokens(opd_teacher, word, text)
head(opd_tidy)
## # A tibble: 6 x 3
## Role Resource word
## <chr> <chr> <chr>
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar into
## 3 Teacher Live Webinar all
## 4 Teacher Live Webinar content
## 5 Teacher Live Webinar areas
## 6 Teacher Live Webinar adn
Note that we also could have just added unnest_tokens(word, text) to our previous piped chain of functions like so:
opd_tidy <- opd_survey %>%
select(Role, Resource, Q21) %>%
rename(text = Q21) %>%
slice(-1, -2) %>%
na.omit() %>%
filter(Role == "Teacher") %>%
unnest_tokens(word, text)
head(opd_tidy)
## # A tibble: 6 x 3
## Role Resource word
## <chr> <chr> <chr>
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar into
## 3 Teacher Live Webinar all
## 4 Teacher Live Webinar content
## 5 Teacher Live Webinar areas
## 6 Teacher Live Webinar adn
There is A LOT to unpack with this function. First notice that unnest_tokens expects a data frame as the first argument, followed by two column names. The first is an output column name that that doesn’t currently exist but will be created as the text is unnested into it (word, in this case). This if followed by the input column that the text comes from which we uncreatively named text. Also notice:
Role and Resource, are retained.to_lower = FALSE argument to turn off this behavior).One final step in tidying our text is to remove words that don’t add much value to our analysis (at least when using this approach) such as “and”, “the”, “of”, “to” etc. The tidytext package contains a stop_words dataset derived from three different lexicons that we’ll use to remove rows that match words in this dataset.
Let’s take a look at these common stop words so we know what we’re getting rid of from our opd_tidy dataset.
head(stop_words)
## # A tibble: 6 x 2
## word lexicon
## <chr> <chr>
## 1 a SMART
## 2 a's SMART
## 3 able SMART
## 4 about SMART
## 5 above SMART
## 6 according SMART
view(stop_words)
In order to remove these stop words, we will use function called anti_join() that looks for matching values in a specific column from two datasets and returns rows from the original dataset that have no matches. For a good overview of the different dplyr joins see here: https://medium.com/the-codehub/beginners-guide-to-using-joins-in-r-682fc9b1f119
Let’s remove rows from our opd_tidy data frame that contain matches in the word column with those in the stop_words dataset and save it as opd_clean since we were done cleaning our data at this point.
opd_clean <- anti_join(opd_tidy, stop_words)
## Joining, by = "word"
head(opd_clean)
## # A tibble: 6 x 3
## Role Resource word
## <chr> <chr> <chr>
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar content
## 3 Teacher Live Webinar adn
## 4 Teacher Live Webinar students
## 5 Teacher Live Webinar move
## 6 Teacher Live Webinar create
anti_join() function in our previous chain that uses the pipe operator? Give it a try and see what happens.anti_join() if we had named the output column from unnest_tokens() “tokens” instead? Hint: Check ?anti_join documentation.As highlighted in both DSEIUR and Learning Analytics Goes to School, calculating summary statistics, data visualization, and feature engineering (the process of creating new variables from a dataset) are a key part of exploratory data analysis. One goal in this phase is explore questions that drove the original analysis and develop new questions and hypotheses to test in later stages. In Section 3, we will calculate some very basic summary statistics from our tidied text, explore key words of interest to gather additional context, and use data visualization to identify patterns and trends that may not be obvious from our tables and numerical summaries. Topics addressed in Section 3 include:
grep package in R, to search for key words among our data set.Prior to making any data visualization, we revisit our or overarching question guiding most of our efforts in this course, “How do we quantify what a text is about?”
In this section, we introduce the following functions:
dplyr functions - count() lets you quickly count the unique values of one or more variables - group_by() takes a data frame and one or more variables to group by - summarise() - mutate() adds new variables and preserves existing ones - left_join() add columns from one dataset to another
tidytext functions - bind_tf_idf() binds the term frequency and inverse document frequency of a tidy text dataset to the dataset
As highlighted in Word Counts are Amazing, one simple but powerful approach to text analysis is counting the frequency in which words occur in a given collection of documents, or corpus.
Now that we have our original survey data in a tidy text format, we can use the count() function from the dplyr package to find the most common words used by teachers when asked, “What was the most beneficial/valuable aspect of this online resource?”
opd_counts <- count(opd_clean, word, sort = TRUE)
# alternatively, we could have use the %>% operator to yield the same result.
opd_counts <- opd_clean %>%
count(word, sort = TRUE)
opd_counts
## # A tibble: 2,268 x 2
## word n
## <chr> <int>
## 1 standards 855
## 2 classroom 793
## 3 core 740
## 4 common 707
## 5 development 590
## 6 teaching 548
## 7 students 512
## 8 professional 470
## 9 resource 424
## 10 understand 382
## # … with 2,258 more rows
Going back to findings from the original report, a strategy as simple basic word counts resulted in key words consistent with findings from the qualitative analysis of focus-group transcripts and open-ended survey responses:
Educators frequently cited that the information and resources provided through the modules improved their understanding of the new standards and the teacher evaluation process.
See also this finding around video clips:
Webinar participants appreciated the useful, updated information presented through a combination of PowerPoint slides and video clips.
One notable distinction between word counts and more traditional qualitative analysis is that broader themes like “convenience” often are not immediately apparent in words counts, but rather emerges from responses containing words like “pace”, “format”, “online”, “ease”, and “access”.
A common theme from focus groups and open-ended survey responses was the convenience of online professional development. One teacher in a focus group stated, “I liked the format. And the way that it was given, it was at your own pace, which works well for our schedules…”
The count() function can also be used with more than one column to count the frequency a word occurs for a select Resource in our dataset.
opd_resource_counts <- opd_clean %>%
count(Resource, word, sort = TRUE)
view(opd_resource_counts)
In this case, we see that “information” was the most common word for Online Learning Modules but did not even make the top 5 for Recorded Webinar:
One common approach to facilitate comparison across documents or groups of text, in our case responses by Online Resource type, is by looking at the frequency that each word occurs among all words for that document group. This also helps to better gauge how prominent the same word is across different groups.
For example, let’s create counts for each Resource and word paring, and then create a new column using the mutate() function that calculations the proportion that word makes up among all words.
To do this a little more efficiently, I’m going to use the %>% operator:
opd_frequencies <- opd_clean %>%
count(Resource, word, sort = TRUE) %>%
group_by(Resource) %>%
mutate(proportion = n / sum(n))
opd_frequencies
## # A tibble: 3,808 x 4
## # Groups: Resource [10]
## Resource word n proportion
## <chr> <chr> <int> <dbl>
## 1 Online Learning Module (e.g. Call for Change, Und… standards 777 0.0378
## 2 Online Learning Module (e.g. Call for Change, Und… classroom 665 0.0323
## 3 Online Learning Module (e.g. Call for Change, Und… core 662 0.0322
## 4 Online Learning Module (e.g. Call for Change, Und… common 585 0.0285
## 5 Online Learning Module (e.g. Call for Change, Und… developm… 513 0.0250
## 6 Online Learning Module (e.g. Call for Change, Und… teaching 484 0.0235
## 7 Online Learning Module (e.g. Call for Change, Und… professi… 412 0.0200
## 8 Online Learning Module (e.g. Call for Change, Und… students 362 0.0176
## 9 Online Learning Module (e.g. Call for Change, Und… resource 346 0.0168
## 10 Online Learning Module (e.g. Call for Change, Und… understa… 336 0.0163
## # … with 3,798 more rows
Using the view() function we can see that “information” makes up about 2.3% of words in responses about the Online Modules, and about 1.7% for Recorded Webinars.
Term frequency-inverse document frequency (tf-idf) is an approach that takes this approach one step further.
As noted in Tidy Text Mining with R:
The statistic tf-idf is intended to measure how important a word is to a document in a collection (or corpus) of documents, for example, to one novel in a collection of novels or to one website in a collection of websites.
Silge and Robinson note that, “The idea of tf-idf is to find the important words for the content of each document by decreasing the weight for commonly used words and increasing the weight for words that are not used very much in a collection or corpus of document… That is, tf-idf attempts to find the words that are important (i.e., common) in a text, but not too common.”
The tidytext package has a function called bind_tf_idf() that takes a tidy text dataset as input with one row per token (term), per document. One column (word here) contains the terms/tokens, one column contains the documents (book in this case), and the last necessary column contains the counts, how many times each document contains each term (n in this example).
Because tf-idf can account through weighting for “too common” words like “and” or “but”, when calculating tf-idf it is not necessary to remove stop words. However, we will need add a column for total words for each Resource type which can be accomplished in a couples steps.
First, let’s recycle our opd_teacher data frame and calculate counts for each word again, but this time instead of word counts for the total data set, we’ll calculate word counts for each ‘Resource’.
opd_words <- opd_teacher %>%
unnest_tokens(word, text) %>%
count(Resource, word, sort = TRUE)
head(opd_words)
## # A tibble: 6 x 3
## Resource word n
## <chr> <chr> <int>
## 1 Online Learning Module (e.g. Call for Change, Understanding the S… to 3336
## 2 Online Learning Module (e.g. Call for Change, Understanding the S… the 2272
## 3 Online Learning Module (e.g. Call for Change, Understanding the S… my 1555
## 4 Online Learning Module (e.g. Call for Change, Understanding the S… in 1374
## 5 Online Learning Module (e.g. Call for Change, Understanding the S… i 1175
## 6 Online Learning Module (e.g. Call for Change, Understanding the S… and 1129
Next, let’s calculate the total words per Resource type:
total_words <- opd_words %>%
group_by(Resource) %>%
summarise(total = sum(n))
## `summarise()` ungrouping output (override with `.groups` argument)
total_words
## # A tibble: 10 x 2
## Resource total
## <chr> <int>
## 1 Calendar 171
## 2 Document, please specify (i.e. Facilitator's Guide, Crosswalks, Sample… 521
## 3 Live Webinar 81
## 4 Online Learning Module (e.g. Call for Change, Understanding the Standa… 47196
## 5 Other, please specify 3773
## 6 Promotional Video 233
## 7 Recorded Webinar or Presentation (e.g. Strategic Staffing, Standards a… 1212
## 8 Summer Institute/RESA PowerPoint Presentations 1173
## 9 Website, please specify 2139
## 10 Wiki 1244
Now let’s append the total column from total_words to our opd_words data frame:
opd_totals <- left_join(opd_words, total_words)
## Joining, by = "Resource"
opd_totals
## # A tibble: 5,063 x 4
## Resource word n total
## <chr> <chr> <int> <int>
## 1 Online Learning Module (e.g. Call for Change, Understand… to 3336 47196
## 2 Online Learning Module (e.g. Call for Change, Understand… the 2272 47196
## 3 Online Learning Module (e.g. Call for Change, Understand… my 1555 47196
## 4 Online Learning Module (e.g. Call for Change, Understand… in 1374 47196
## 5 Online Learning Module (e.g. Call for Change, Understand… i 1175 47196
## 6 Online Learning Module (e.g. Call for Change, Understand… and 1129 47196
## 7 Online Learning Module (e.g. Call for Change, Understand… for 1099 47196
## 8 Online Learning Module (e.g. Call for Change, Understand… of 859 47196
## 9 Online Learning Module (e.g. Call for Change, Understand… a 795 47196
## 10 Online Learning Module (e.g. Call for Change, Understand… standa… 777 47196
## # … with 5,053 more rows
Finally, we’re ready to use the bind_tf_idf() function to calculate a tf-idf statistic for each word and assess it’s relative importance to a given online resource type:
opd_tf_idf <- opd_totals %>%
bind_tf_idf(word, Resource, n)
opd_tf_idf
## # A tibble: 5,063 x 7
## Resource word n total tf idf tf_idf
## <chr> <chr> <int> <int> <dbl> <dbl> <dbl>
## 1 Online Learning Module (e.g. Call fo… to 3336 47196 0.0707 0 0
## 2 Online Learning Module (e.g. Call fo… the 2272 47196 0.0481 0 0
## 3 Online Learning Module (e.g. Call fo… my 1555 47196 0.0329 0 0
## 4 Online Learning Module (e.g. Call fo… in 1374 47196 0.0291 0 0
## 5 Online Learning Module (e.g. Call fo… i 1175 47196 0.0249 0 0
## 6 Online Learning Module (e.g. Call fo… and 1129 47196 0.0239 0 0
## 7 Online Learning Module (e.g. Call fo… for 1099 47196 0.0233 0 0
## 8 Online Learning Module (e.g. Call fo… of 859 47196 0.0182 0 0
## 9 Online Learning Module (e.g. Call fo… a 795 47196 0.0168 0 0
## 10 Online Learning Module (e.g. Call fo… stand… 777 47196 0.0165 0.105 0.00173
## # … with 5,053 more rows
view(opd_tf_idf)
Notice that idf and thus tf-idf are zero for these extremely common words (typically stop words). These are all words that appear in teacher responses for all online resource types, so the idf term (which will then be the natural log of 1) is zero. The inverse document frequency (and thus tf-idf) is very low (near zero) for words that occur in many of the documents in a collection; this is how this approach decreases the weight for common words. The inverse document frequency will be a higher number for words that occur in fewer of the documents in the collection.
On one final note, while it has proved useful in text mining, search engines, etc., its theoretical foundations are considered less than firm by information theory experts…"
In the next section, we’ll use some data visualization strategies to help us interpret and find patterns in these rather dense output tables.
opd_resource_counts and searching in the source how, how might you use the filter() function to find return the most common words for Recorded Webinars?opd_tf_idf data frame we created?opd_benefits data frame. For frequencies and tf-idf, group by Role instead of Resource.This section is a really quick aside and primarily meant to introduce the grep package that we’ll be using in future units.
A quick word count actually resulted in findings fairly consistent with some of the qualitative findings reported, but also lacked some nuance, unsurprisingly, and left some questions about what some of the more frequent words were in reference to.
Let’s use our reduced opd_teacher survey data frame that contains the complete teacher responses and use the handy filter(), select() and grepl() function to select just our text column and filter out responses that contain key words of interest. For example, what aspects of “online” made it beneficial.
We can view all quotes in the source pane, or use the sample_n(), yes from the dplyr package, to select any number of random quotes. In this case 20:
opd_quotes <- opd_teacher %>%
select(text) %>%
filter(grepl('online', text))
view(opd_quotes)
sample_n(opd_quotes, 20)
## # A tibble: 20 x 1
## text
## <chr>
## 1 "As an online professional development"
## 2 "online to keep up with current standards"
## 3 "I'm still evaluating how this will help my classroom. I feel that I might …
## 4 "online to make lesson plans"
## 5 "I will use the revised Bloom's taxonomy charts provided in the online to he…
## 6 "To complete an online learning module."
## 7 "I am not using it much, since it is not very practical or very informative …
## 8 "online"
## 9 "online"
## 10 "I do not learn well on computer, so it is difficult to not have the ability…
## 11 "Use the online module to reflect on creation of \"I can \" statements and t…
## 12 "I am using the online modules to learn more of the expectations for teacher…
## 13 "online"
## 14 "online"
## 15 "We are doing PD online."
## 16 "On online"
## 17 "As a support person in my school I am providing the information that I lear…
## 18 "I am using a number of the resources that are online."
## 19 "online module"
## 20 "To support innovations in my creation of new lessons promoting self paced …
In some cases, we can see that the use of the word “online” was simply repetition of the question prompt, but in other cases we can see that it’s associated with the broader theme of “convenience” as with the quote, “This online resources gave me the opportunity to study on my own time.”
Note that you can also use regular express operators with grep like the * operator to search for word stems. For example using inform* in our search will return quotes with “inform”, “informative”, “information”, etc.
opd_quotes <- opd_teacher %>%
select(text) %>%
filter(grepl('inform*', text))
view(opd_quotes)
sample_n(opd_quotes, 20)
## # A tibble: 20 x 1
## text
## <chr>
## 1 This resource has supplemented prevoius knowledge of standard based planning…
## 2 Learning more about this information.
## 3 To become more informed.
## 4 After we understand the new standards, one of the new challenges as we trans…
## 5 I plan to use the information for my upcoming and future research projects i…
## 6 As Grade Level Chair, I have used the information in planning PLC and for Co…
## 7 Viewing online and will apply the information I learned as I plan and teach …
## 8 information
## 9 I am required to take these modules. I really do listen/read the information.
## 10 I am sharing the information in my PLC's.
## 11 new information
## 12 As a support person in my school I am providing the information that I learn…
## 13 I will be implementing the information learned during the scoring for the So…
## 14 Too much info at one time and online is not very helpful. A small booklet w…
## 15 The resource is being used on a individual basis. With the information we ar…
## 16 I'm using this resource to inform myself about the elements of the new Commo…
## 17 To gain information on Revised Bloom's Taxonomy
## 18 I use this resource as information to better instruct my students and gain a…
## 19 gathering information to use for the upcoming school year 2012-2013, using c…
## 20 I will use this information as a way to better evaluate and record student p…
We covered data visualization pretty extensively in ECI 586: Introduction to Learning Analytics, but for those new to data visualization in R, the go to package for standard charts and graphs is ggplot2. Hadley Wickham’s R for Data Science and [ggplot2: Elegant Graphics for Data] are also great introductions to data visualization in R with ggplot2.
The wordcloud2 packages is pretty dead simple for generating HTML based word clouds.
For example, let’s load our installed wordclouds2 library, and run the wordcloud2() function on our opd_counts data frame:
library(wordcloud2)
wordcloud2(opd_counts)
I use wordclouds pretty sparingly in evaluation reports, but typically include them for open ended items in online Qualtrics survey reports to provide education partners I work with a quick snapshot of the response.
Once installed, I recommend using ?wordclouds2 to view the various arguments for cleaning up the default view.
The bar chart is the workhorse for data viz and is pretty effective for comparing two or more values. Given the unique aspect of our tidy text data frame, however, we are looking at upwards of over 5,000 values (i.e. words and their counts) to compare with our opd_counts data frame and will need some way to limit the number of words to display.
opd_counts %>%
filter(n > 500) %>% # keep rows with word counts greater than 500
mutate(word = reorder(word, n)) %>% #reorder the word variable by n and replace with new variable called word
ggplot(aes(n, word)) + # create a plot with n on x axis and word on y axis
geom_col() # make it a bar plot
#### Small Multiples
Word clouds and bar charts are pretty effective for highlighting the most common words in an entire corpus, or in our case, all teacher survey responses, regarless of resource type being reviewed.
One limitation we ran into earlier when we started looking at word frequencies and tf-idf stats was that it was difficult to easily compare the most common or unique words for each resource type. That is where small multiples come. A small multiple is basically a series of similar graphs or charts using the same scale and axes that make it easier to compare across different document collections of interest, in our case, word counts by resource type.
Let’s use the example illustrated in Text Mining with R to create a small multiple for our opd_frequencies data set instead of the opd_tf_idf
library(forcats)
opd_frequencies %>%
filter(Resource != "Calendar") %>% # remove Calendar responses, too few.
group_by(Resource) %>%
slice_max(proportion, n = 5) %>%
ungroup() %>%
ggplot(aes(proportion, fct_reorder(word, proportion), fill = Resource)) +
geom_col(show.legend = FALSE) +
facet_wrap(~Resource, ncol = 3, scales = "free")
opd_benefits data.